SUHASH REDDY IMMAREDDY - 45693242 - DATA-SCIENCE - PORTFOLIO-1¶
Firstly we need to import the packages inorder to Visualize, Analysing, Modify & Clean the datasets¶
In [1]:
import os
import matplotlib as mp
import matplotlib.pyplot as plt#-------------------------Importing pyplot as plt
import pandas as pd
from matplotlib import pyplot as plt#--------------------Importing pyplot as plt ------ Can write in this fashion too.
from datetime import timedelta
import seaborn as sns
plt.style.use('seaborn')
%matplotlib inline
In [2]:
os.getcwd() # show's the current working directory
Out[2]:
In [3]:
os.chdir("data")
print("""changing the directory to the location where the data-set is available so that now we can read the file""")
os.getcwd()
Out[3]:
Importing Strava.csv¶
In [4]:
strava = pd.read_csv('strava_export.csv', index_col='date', parse_dates=True) #making the date column as index
print(strava.head(2))
we had read the strava file and made the date column as the index and then localizing the time zones¶
In [5]:
strava.index = strava.index.tz_convert('Australia/Sydney')
strava.head()
Out[5]:
In [6]:
strava.shape
Out[6]:
Importing Cheetah.csv¶
In [7]:
cheetah = pd.read_csv('cheetah.csv', skipinitialspace=True)
cheetah.head()
Out[7]:
In [8]:
cheetah.index = pd.to_datetime(cheetah['date'] + ' ' + cheetah['time'])
cheetah.index = cheetah.index.tz_localize('Australia/Sydney')
cheetah.head()
Out[8]:
we had read the cheetah file and made the date column as the index and then localizing the time zones¶
WE now try to understand the type of data that is present in the strava and cheetah file.¶
In [9]:
print("The shape of strava table is rows, column",strava.shape)
print(strava.get_dtype_counts())
print("\n")
print("getting the datatype for all columns", strava.dtypes)
In [10]:
print("The shape of Cheetah table is rows, column",cheetah.shape)
print(cheetah.get_dtype_counts())
print("\n")
print("getting the datatype for all columns", cheetah.dtypes)
we are doing inner join so that we will get the records that present in both dataframes which are strava and cheetah¶
In [11]:
cheetah_strava_innerjoin = pd.merge(left = cheetah, right = strava, left_index = True, right_index= True, how='inner')
print("The new shape of the joined table is :",cheetah_strava_innerjoin.shape)
cheetah_strava_innerjoin.head()
Out[11]:
In [12]:
print("cheeta_strava_innerjoin-count---------------------------------------------------------------")
total_row_count = cheetah_strava_innerjoin.count()
print(total_row_count.head(5))
print("strava-count--------------------------------------------------------------------------------")
total_row_count1 = strava.count()
print(total_row_count1.head(5))
print("cheetah-count-------------------------------------------------------------------------------")
total_row_count2 = cheetah.count()
print(total_row_count2.head(5))
So after joining the two tables on index, we got the new table which now only contains the data that is common in both data sets. We can also see that some of the rows were taken off from the joined table compared to the respective individual tables where we can see rows with a count greater than 243.¶
1. Remove rides with no measured power (where device_watts is False) - these are commutes or MTB rides¶
In [13]:
cheetah_strava_innerjoin['device_watts'].head()
Out[13]:
were are trying to Know the type of data holds by the device_watts column , Which is Boolean i,e. True OR False, so now we can drop the rows which contains the value as False in the device_watts column¶
In [14]:
cheetah_strava_innerjoin = cheetah_strava_innerjoin[cheetah_strava_innerjoin.device_watts != False]
In [15]:
print("The Shape of the joined table after droping rows where device_watts is False", cheetah_strava_innerjoin.shape)
After droping the rows where the device_watts if False the row count is dropped to 209 from 243¶
2. Look at the distributions of some key variables: time, distance, average speed, average power, TSS. Are they normally distributed? Skewed?¶
There are four varibales with the name time so we will be plotting the distribution graphs for all of those variables¶
In [16]:
import sys
import warnings
if not sys.warnoptions:
warnings.simplefilter("ignore")
cheetah_strava_innerjoin['time'] = pd.to_datetime(cheetah_strava_innerjoin['time']).dt.strftime('%H:%M:%S')
sns.scatterplot(cheetah_strava_innerjoin['time'],cheetah_strava_innerjoin.index)
plt.show()
Ploting the time against the index which is date-time we can see that it closely follows a straight line(can exclude the outliers), so that we can say that time column follows the normal distribution¶
In [17]:
def skewness(x):
res = 0
m = x.mean()
s = x.std()
for i in x:
res += (i-m) * (i-m) * (i-m)
res /= ( len(x) * s * s * s)
return res
In [18]:
sns.distplot(cheetah_strava_innerjoin.moving_time)
print("Skewness of the moving_time = ",skewness(cheetah_strava_innerjoin.moving_time))
The Distribution of the moving time is bimodal distribution and has a positive skewness which means it is skewed right.¶
In [19]:
sns.distplot(cheetah_strava_innerjoin.elapsed_time)
print("Skewness of the elapsed_time = ",skewness(cheetah_strava_innerjoin.elapsed_time))
The Distribution of the elapsed time is bimodal distribution and has a positive skewness which means it is skewed right.¶
In [20]:
sns.distplot(cheetah_strava_innerjoin['Time Moving'])
print("Skewness of the Time Moving = ",skewness(cheetah_strava_innerjoin['Time Moving']))
The Distribution of the time moving is bimodal distribution and has a positive skewness which means it is skewed right.¶
In [21]:
sns.distplot(cheetah_strava_innerjoin.distance)
print("Skewness of the distance = ",skewness(cheetah_strava_innerjoin.distance))
The Distribution of the distance is bimodal distribution and has a positive skewness which means it is skewed right.¶
In [22]:
sns.distplot(cheetah_strava_innerjoin['Average Power'])
print("Skewness of the Average Power = ",skewness(cheetah_strava_innerjoin['Average Power']))
The Distribution of theAverage Power is bimodal distribution and has a negative skewness which means it is skewed left.¶
In [23]:
cheetah_strava_innerjoin['average_watts'] = cheetah_strava_innerjoin['average_watts'].fillna(0.0)
sns.distplot(cheetah_strava_innerjoin.average_watts)
print("Skewness of the Average watts = ",skewness(cheetah_strava_innerjoin['average_watts']))
The Distribution of the average_watts is bimodal distribution and has a negative skewness which means it is skewed left, it is mainly because of the outlier if we exclude that one then it follows a normal distribution.¶
In [24]:
sns.distplot(cheetah_strava_innerjoin.TSS)
print("Skewness of the TSS = ",skewness(cheetah_strava_innerjoin['TSS']))
The Distribution of theTss is bimodal distribution and has a positive skewness which means it is skewed right.¶
In [25]:
sns.distplot(cheetah_strava_innerjoin['Average Speed'])
print("Skewness of the Average Speed = ",skewness(cheetah_strava_innerjoin['Average Speed']))
The Distribution of the moving time is multimodal distribution and had a Negative skewness which means it is skewed left, if we exclude the outlier then it follows bimodal distribution.¶
3. Explore the relationships between the following variables. Are any of them corrolated with each other (do they vary together in a predictable way)? Can you explain any relationships you observe?¶
Distance¶
Moving Time¶
Average Speed¶
Heart Rate¶
Power (watts)¶
Normalised power (NP)¶
Training Stress Score¶
Elevation Gain¶
In [26]:
C_S_I = cheetah_strava_innerjoin[["distance","moving_time","Average Speed","average_heartrate","Average Power","NP","TSS","Elevation Gain"]]
In [27]:
C_S_I.head()
Out[27]:
In [28]:
import sys
import warnings
if not sys.warnoptions:
warnings.simplefilter("ignore")
sns.pairplot(C_S_I)
Out[28]:
In [29]:
CORRELATION = C_S_I.corr()
In [30]:
CORRELATION
Out[30]:
From the graph and the correlation matrix we can see that : -¶
With respect to distance -> it is higly correlated with moving_time followed by TSS and Elevation Gain.¶
With respect to moving_time -> it is highly correlated with distance followed by TSS and Elevation Gain.¶
With respect to average speed ->it is higly correlated with Average Power followed by average heart rate.¶
With respect to average_heartrate -> it is higly crrelated with Average Speed followed by Average heartrate.¶
With respect to average power -> it is higly correlated with NP followed by Average Speed.¶
With respect to NP -> it is higly correlated with Average Power.¶
With respect toTSS -> it is higly correlated with distance followed by moving_time and Elevatio Gain.¶
With respect to Elevation gain -> it is higly correlated with TSS followed by moving_time and distance.¶
4. We want to explore the differences between the three categories: Race, Workout and Ride.¶
4a. Use scatter plots with different colours for each category to explore how these categories differ.¶
4b. Use histograms or box plots to visualise the different distributions of a variable for the three categories.¶
4c. In both cases, experiment with different variables but only include those that are interesting in your final notebook (if none are interesting, show us a representative example).¶
In [31]:
C_S_I1 = cheetah_strava_innerjoin[["distance","workout_type","moving_time","Average Speed","average_heartrate","Average Power","NP","TSS","Elevation Gain"]]
In [32]:
C_S_I1.head()
Out[32]:
Swarm Plots tellls us about the data distribution for each category in a variable.¶
In [33]:
sns.catplot(x='workout_type',y='average_heartrate',kind='swarm',data=C_S_I1)
plt.grid(True)
In [34]:
sns.catplot(x='workout_type',y='average_heartrate',kind='box',data=C_S_I1)
plt.grid(True)
In [35]:
sns.catplot(x='workout_type',y='distance',kind='swarm',data=C_S_I1)
plt.grid(True)
In [36]:
sns.catplot(x='workout_type',y='distance',kind='box',data=C_S_I1)
plt.grid(True)
In [37]:
#hue='workout_type'
import sys
import warnings
if not sys.warnoptions:
warnings.simplefilter("ignore")
sns.pairplot(C_S_I1, hue = 'workout_type')
Out[37]:
distance -> moving_time, TSS and Elevation Gain (Ride increases with increase in distance)¶
moving_time -> distance, TSS and Elevation Gain (Ride increases with increase in moving_time)¶
average_speed -> avg_heartrate, average power, NP (Race increases with increase in average speed)¶
average_heartrate -> avg_speed, avg_power, NPP (both Race and RIde increases with increase in average_heartrate)¶
TSS -> elevation_gain (Ride increses with increse in TSS)¶
Elevation Gain -> Ride increase with increase in Elevation gain against all other variables¶
In [38]:
#what leads to more kudos
C_S_I2 = cheetah_strava_innerjoin[["workout_type","kudos","distance","moving_time"]]
sns.catplot(x="workout_type", y="kudos", hue="workout_type", kind="box", data=C_S_I2)
Out[38]:
In [39]:
sns.catplot(x="workout_type", y="kudos", hue="workout_type", kind="bar", data=C_S_I2)
Out[39]:
well we can see from the boxplot as well as bar chart that Race workout_type leads to More Kudos¶
In [40]:
C_S_I1 = cheetah_strava_innerjoin[["distance","workout_type","kudos","moving_time","Average Speed","average_heartrate","Average Power","NP","TSS","Elevation Gain"]]
In [41]:
CORRELATION = C_S_I1.corr()
In [42]:
CORRELATION
Out[42]:
In [43]:
sns.pairplot(C_S_I2, hue = 'workout_type')
Out[43]:
In [44]:
CORRELATION2 = C_S_I2.corr()
In [45]:
CORRELATION2
Out[45]:
From the correltion matrix1 we can see that distance explains the more about kudos than compared with moving_time and TSS, so we are including only distance and moving time in my final report and implemented the paitplot.¶
As the no of kudos increases with the increase in distance and TSS, but distance is the main variable¶
Challenge 2¶
In [46]:
suhash = pd.DataFrame()
In [47]:
suhash["date"] = cheetah_strava_innerjoin.date
In [48]:
suhash["distance"] = cheetah_strava_innerjoin.distance
In [49]:
suhash["TSS"] = cheetah_strava_innerjoin.TSS
In [50]:
suhash["AverageSpeed"] = cheetah_strava_innerjoin["Average Speed"]
In [51]:
suhash.head()
Out[51]:
In [52]:
from datetime import datetime
from datetime import timedelta
In [53]:
suhash['date'] = pd.to_datetime(suhash['date'])
In [54]:
suhash.head()
Out[54]:
In [55]:
suhash1 = suhash.reset_index(drop=True)
In [56]:
suhash1.head()
Out[56]:
In [57]:
Distance_Month = suhash1.set_index('date').groupby(pd.Grouper(freq='M'))['distance','TSS'].sum()
In [58]:
Distance_Month["Averageg_Speed"] = suhash1.set_index('date').groupby(pd.Grouper(freq='M'))['AverageSpeed'].mean()
In [59]:
Distance_Month = Distance_Month.reset_index()
In [60]:
Distance_Month
Out[60]:
In [61]:
plt.figure(figsize = (10,5))
plt.plot(Distance_Month["date"], Distance_Month["distance"], marker = '*')
plt.title("Distance Travelled by month")
Out[61]:
In [ ]:
SUHASH REDDY IMMAREDDY - 45693242 - DATA-SCIENCE - PORTFOLIO-2¶
In [62]:
import os
In [65]:
os.getcwd()
Out[65]:
In [70]:
os.chdir("C:\\Users\\suhas\\Documents\\GitHub\\portfolio-2019-suhashimmareddy\\Appliances-energy-prediction-data")
In [71]:
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pylab as plt
from matplotlib.colors import ListedColormap
%matplotlib inline
from sklearn.model_selection import train_test_split
from sklearn import linear_model
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error,r2_score
from sklearn.metrics import confusion_matrix, accuracy_score
from sklearn.feature_selection import RFE
from datetime import datetime
from datetime import timedelta
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
import seaborn as sns
%matplotlib inline
from sklearn.feature_selection import f_regression
from sklearn.preprocessing import MinMaxScaler
from sklearn.ensemble import RandomForestRegressor
reading the data and understanding the the structure¶
In [72]:
edc = pd.read_csv('energydata_complete.csv')
In [73]:
edc.head()
Out[73]:
In [74]:
edc.shape
Out[74]:
In [75]:
edc.describe()
Out[75]:
In [76]:
edc.columns
Out[76]:
In [77]:
edc.dtypes
Out[77]:
Data exploration - we try to add some new columns for better understandings¶
In [78]:
edc1 = edc # just making sure that we do have a backup file of the original one
In [79]:
edc1['date'] = pd.to_datetime(edc1['date'])
In [80]:
len(edc1)
Out[80]:
In [81]:
import sys
import warnings
if not sys.warnoptions:
warnings.simplefilter("ignore")
month = []
hours = []
week = []
Day = []
for i in range (len(edc1)):
month.append(edc1['date'][i].month_name())
hours.append(edc1['date'][i])
week.append(edc1['date'][i].week)
Day.append(edc1['date'][i].dayofweek)
In [82]:
a = set(month)
In [83]:
a
Out[83]:
In [84]:
#Convert to Series
A=pd.Series(month)
B=pd.Series(hours).dt.floor("H") #floor the hour
C=pd.Series(week)
D=pd.Series(Day)
#adding the series to the dataframe
edc1['Month']=A
edc1['Hour']=B
edc1['week']=C
edc1['Day']=D
edc1['Time'] = edc1.Hour.dt.hour
In [85]:
edc1.columns
Out[85]:
In [86]:
edc1.head()
Out[86]:
In [87]:
edc1.shape
Out[87]:
In [88]:
b = set(Day)
In [89]:
b
Out[89]:
0 is Monday, 1 is tuesday, 2 is wednesday, 3 is thursday, 4 is friday, 5 is saturday, 6 is sunday¶
In [90]:
plt.figure(figsize=(20, 7))
sns.distplot(edc1.Appliances)
Out[90]:
In [91]:
plt.figure(figsize=(20, 7))
plt.hist(edc1["Appliances"], bins='auto', color='#ff0000', alpha=0.7, rwidth=0.85)
plt.show()
In [92]:
plt.figure(figsize=(16, 6))
sns.set(style="whitegrid")
sns.boxplot(edc1["Appliances"])
Out[92]:
In [93]:
# box plot of Appliances energy Consuption, grouped by month
plt.figure(figsize=(20, 7))
sns.boxplot(x="Appliances", y="Month", data=edc1)
Out[93]:
In [94]:
plt.figure(figsize=(20, 7))
ax1 = sns.lineplot(data=edc1["Appliances"], color="coral", label="line")
ax1.set_ylabel('Appliances (Wh)');
In [95]:
plt.figure(figsize=(20, 7))
plt.ylabel('Appliances(wh)')
plt.xlabel('per month')
plt.plot(edc1.date,edc1.Appliances)
plt.xticks(rotation = -75)
Out[95]:
In [96]:
week = edc1["date"].dt.week
first_week = edc1[week == min(week)]
first_week
plt.figure(figsize=(20,7))
plt.plot(first_week['date'],first_week['Appliances'])
Out[96]:
Now we try to represent Hourly energy consumption of appliances in heat map for four consecutive weeks[Week3,Week4,Week5,Week6]¶
In [97]:
energy1=pd.DataFrame() # making a new dataframe and appending alll required feilds to it that are required for the heat map
energy2 =edc1.groupby('Hour',as_index=False).agg({"Appliances": "sum"})
In [98]:
energy2.head()
Out[98]:
In [99]:
energy1['Hour'] = edc1['Hour']
energy1['week'] = edc1['week']
energy1['Day'] = edc1['Day']
energy1['Time'] = edc1['Time']
In [100]:
energy1.head()
Out[100]:
In [101]:
energy3 =energy1.groupby('Hour',as_index=False).first()
In [102]:
energy3.head()
Out[102]:
In [103]:
energy1=pd.merge(energy3,energy2)
In [104]:
energy1.head()
Out[104]:
In [105]:
week3=energy1[energy1.week == 3]
week3 = week3.drop(['Hour', 'week'], axis=1) #selecting only the required columns
In [106]:
week3.head()
Out[106]:
In [107]:
week3 = week3.pivot("Time","Day","Appliances")
week4=energy1[energy1.week == 4]
week4 = week4.drop(['Hour', 'week'], axis=1)
week4 = week4.pivot("Time","Day","Appliances")
week5=energy1[energy1.week == 5]
week5 = week5.drop(['Hour', 'week'], axis=1)
week5 = week5.pivot("Time","Day","Appliances")
week6=energy1[energy1.week == 6]
week6 = week6.drop(['Hour', 'week'], axis=1)
week6 = week6.pivot("Time","Day","Appliances")
In [108]:
plt.figure(figsize=(4, 9))
plt.title("Week-3 Heat Map")
ax = sns.heatmap(week3,cmap = "YlOrRd",annot=True, fmt="d", linewidths=.5)
In [109]:
plt.figure(figsize=(4, 9))
plt.title("Week-4 Heat Map")
ax = sns.heatmap(week4,cmap = "YlOrRd",annot=True, fmt="d", linewidths=.5)
In [110]:
plt.figure(figsize=(4, 9))
plt.title("Week-5 Heat Map")
ax = sns.heatmap(week5,cmap = "YlOrRd",annot=True, fmt="d", linewidths=.5)
In [111]:
plt.figure(figsize=(4, 9))
plt.title("Week-6 Heat Map")
ax = sns.heatmap(week6,cmap = "YlOrRd",annot=True, fmt="d", linewidths=.5)
In the research Paper they did the heat map on top of the training data but i had done it on the total data¶
In [112]:
edc1.columns
Out[112]:
In [113]:
## each variable distribution and the correlation value with respect appliances is given below
In [114]:
def corr(x, y, **kwargs):
coef = np.corrcoef(x, y)[0][1]
label = r'$\rho$ = ' + str(round(coef, 2))
ax = plt.gca()
ax.annotate(label, xy = (0.2, 0.95), size = 20, xycoords = ax.transAxes)
In [115]:
ss = edc1[['Appliances','lights', 'T1', 'RH_1', 'T2', 'RH_2', 'T3','RH_3' ]]
In [116]:
g = sns.pairplot(ss)
g.map_lower(corr)
g.map_upper(corr)
plt.show()
In [117]:
ss1 = edc1[['Appliances','T4', 'RH_4', 'T5', 'RH_5', 'T6', 'RH_6', 'T7']]
In [118]:
g = sns.pairplot(ss1)
g.map_lower(corr)
g.map_upper(corr)
plt.show()
In [119]:
ss2 = edc1[['Appliances','RH_7', 'T8','RH_8', 'T9', 'RH_9', 'T_out', 'Press_mm_hg']]
In [120]:
g = sns.pairplot(ss2)
g.map_lower(corr)
g.map_upper(corr)
plt.show()
In [121]:
ss3 = edc1[['Appliances','RH_out', 'Windspeed','Visibility', 'Tdewpoint', 'rv1', 'rv2']]
In [122]:
g = sns.pairplot(ss3)
g.map_lower(corr)
g.map_upper(corr)
plt.show()
In [123]:
# model validation
In [124]:
train = pd.read_csv('training.csv', index_col='date', parse_dates=True)
In [125]:
test = pd.read_csv('testing.csv', index_col='date', parse_dates=True)
In [126]:
train = train.join(pd.get_dummies(train['Day_of_week']))
train.head()
Out[126]:
In [127]:
train = train.join(pd.get_dummies(train['WeekStatus']))
train.head()
Out[127]:
In [128]:
test = test.join(pd.get_dummies(test['Day_of_week']))
test.head()
Out[128]:
In [129]:
test = test.join(pd.get_dummies(test['WeekStatus']))
test.head()
Out[129]:
In [130]:
### traing the dataset with train dataset and validating against test dataset
In [131]:
c = train.columns
In [132]:
c = c.drop(['Day_of_week','WeekStatus','Weekend'])
In [133]:
c
Out[133]:
In [134]:
X_train = train.drop(['Appliances','Day_of_week','WeekStatus'], axis = 1)
Y_train = train['Appliances']
X_test = test.drop(['Appliances','Day_of_week','WeekStatus'], axis = 1)
Y_test = test['Appliances']
print(X_train.shape)
print(Y_train.shape)
print(X_test.shape)
print(Y_test.shape)
In [135]:
### Fitting the linear model
In [136]:
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_error
model = LinearRegression()
model.fit(X_train, Y_train)
Out[136]:
In [137]:
def mean_absolute_percentage_error1(y_true, y_pred):
y_true, y_pred = np.array(y_true), np.array(y_pred)
return np.mean(np.abs((y_true - y_pred) / y_true)) * 100
In [138]:
import math
Y_pred = model.predict(X_test)
print("r2 for test data is",r2_score(Y_test, Y_pred))
print("MSE for test data is",mean_squared_error(Y_test, Y_pred))
print("RMSE for test data is",math.sqrt(mean_squared_error(Y_test, Y_pred)))
print("MAE for test data is",mean_absolute_error(Y_test, Y_pred))
print("MAPE for test data is",mean_absolute_percentage_error1(Y_test, model.predict(X_test)))
In [139]:
residuals = Y_test - Y_pred
plt.figure(figsize = (10,5))
plt.scatter(test.Appliances,residuals)
plt.xlabel("Appliances")
plt.ylabel("residuals")
Out[139]:
In [140]:
Y_pred1 = model.predict(X_train)
print("r2 for train data is",r2_score(Y_train, Y_pred1))
print("MSE for train data is",mean_squared_error(Y_train, Y_pred1))
print("RMSE for train data is",math.sqrt(mean_squared_error(Y_train, Y_pred1)))
print("MAE for train data is",mean_absolute_error(Y_train, Y_pred1))
print("MAPE for train data is",mean_absolute_percentage_error1(Y_train, model.predict(X_train)))
In [141]:
## Fitting the RFE
In [142]:
import sys
import warnings
if not sys.warnoptions:
warnings.simplefilter("ignore")
estimator = LinearRegression()
selector = RFE(estimator, 30, step=1)
selector = selector.fit(X_train, Y_train)
y_predict = selector.predict(X_test)
In [143]:
print("r2 for test data is",r2_score(Y_test, y_predict))
print("MSE for test data is",mean_squared_error(Y_test, y_predict))
print("RMSE for test data is",math.sqrt(mean_squared_error(Y_test, y_predict)))
print("MAE for test data is",mean_absolute_error(Y_test, y_predict))
print("MAPE for test data is",mean_absolute_percentage_error1(Y_test, y_predict))
In [144]:
residuals = Y_test - y_predict
plt.figure(figsize = (10,5))
plt.scatter(test.Appliances,residuals)
plt.xlabel("Appliances")
plt.ylabel("residuals")
Out[144]:
In [145]:
Y_pred1 = selector.predict(X_train)
print("r2 for train data is",r2_score(Y_train, Y_pred1))
print("MSE for train data is",mean_squared_error(Y_train, Y_pred1))
print("RMSE for train data is",math.sqrt(mean_squared_error(Y_train, Y_pred1)))
print("MAE for train data is",mean_absolute_error(Y_train, Y_pred1))
print("MAPE for train data is",mean_absolute_percentage_error1(Y_train, selector.predict(X_train)))
We are getting the same result when were are using the linear regression and Recursive feature extraction with linear model where the MSE value is very high and r2 square values is low so that we can say that the model is not doing a better job.¶
In [146]:
## Fitting a Random Forest Regressor
In [147]:
from sklearn.ensemble import RandomForestRegressor
regressor = RandomForestRegressor(n_estimators = 100, random_state = 0)
regressor.fit(X_train, Y_train)
Out[147]:
In [148]:
Y_pred = regressor.predict(X_test)
print("r2 for test data is",r2_score(Y_test, Y_pred))
print("MSE for test data is",mean_squared_error(Y_test, Y_pred))
print("RMSE for test data is",math.sqrt(mean_squared_error(Y_test, Y_pred)))
print("MAE for test data is",mean_absolute_error(Y_test, Y_pred))
print("MAPE for test data is",mean_absolute_percentage_error1(Y_test, model.predict(X_test)))
In [149]:
Y_pred1 = regressor.predict(X_train)
print("r2 for train data is",r2_score(Y_train, Y_pred1))
print("MSE for train data is",mean_squared_error(Y_train, Y_pred1))
print("RMSE for train data is",math.sqrt(mean_squared_error(Y_train, Y_pred1)))
print("MAE for train data is",mean_absolute_error(Y_train, Y_pred1))
print("MAPE for train data is",mean_absolute_percentage_error1(Y_train, regressor.predict(X_train)))
In [150]:
import sys
import warnings
if not sys.warnoptions:
warnings.simplefilter("ignore")
estimator = RandomForestRegressor(n_estimators = 100, random_state = 0)
selector = RFE(estimator, 30, step=1)
selector = selector.fit(X_train, Y_train)
y_predict = selector.predict(X_test)
In [151]:
print("r2 for test data is",r2_score(Y_test, y_predict))
print("MSE for test data is",mean_squared_error(Y_test, y_predict))
print("RMSE for test data is",math.sqrt(mean_squared_error(Y_test, y_predict)))
print("MAE for test data is",mean_absolute_error(Y_test, y_predict))
print("MAPE for test data is",mean_absolute_percentage_error1(Y_test, y_predict))
In [152]:
Y_pred1 = selector.predict(X_train)
print("r2 for train data is",r2_score(Y_train, Y_pred1))
print("MSE for train data is",mean_squared_error(Y_train, Y_pred1))
print("RMSE for train data is",math.sqrt(mean_squared_error(Y_train, Y_pred1)))
print("MAE for train data is",mean_absolute_error(Y_train, Y_pred1))
print("MAPE for train data is",mean_absolute_percentage_error1(Y_train, selector.predict(X_train)))
We are getting the same result when were are using the Random Forest and Random forest with Recursive feature extraction where the MSE value is Low and r2 square values is High so that we can say that the model is doing a better job.¶
In [ ]:
We now try to get the Varible importance for the Random Forest¶
In [153]:
ranks = {}
# Create our function which stores the feature rankings to the ranks dictionary
def ranking(ranks, names, order=1):
minmax = MinMaxScaler()
ranks = minmax.fit_transform(order*np.array([ranks]).T).T[0]
ranks = map(lambda x: round(x,2), ranks)
return dict(zip(names, ranks))
In [154]:
lr = LinearRegression(normalize=True)
lr.fit(X_train,Y_train)
#stop the search when only the last feature is left
rfe = RFE(lr, n_features_to_select=1, verbose =3 )
rfe.fit(X_train,Y_train)
ranks["RFE"] = ranking(list(map(float, rfe.ranking_)), c, order=-1)
In [155]:
rf = RandomForestRegressor(n_jobs=-1, n_estimators=50, verbose=3)
rf.fit(X_train,Y_train)
ranks["RF"] = ranking(rf.feature_importances_, c);
In [156]:
r = {}
for name in c:
r[name] = round(np.mean([ranks[method][name] for method in ranks.keys()]), 2)
methods = sorted(ranks.keys())
ranks["Mean"] = r
methods.append("Mean")
print("\t%s" % "\t".join(methods))
for name in c:
print("%s\t%s" % (name, "\t".join(map(str,
[ranks[method][name] for method in methods]))))
In [157]:
# Put the mean scores into a Pandas dataframe
meanplot = pd.DataFrame(list(r.items()), columns= ['Feature','Mean Ranking'])
# Sort the dataframe
meanplot = meanplot.sort_values('Mean Ranking', ascending=False)
In [158]:
# Let's plot the ranking of the features
sns.factorplot(x="Mean Ranking", y="Feature", data = meanplot, kind="bar",
size=14, aspect=1.9, palette='coolwarm')
Out[158]:
In [ ]:
SUHASH REDDY IMMAREDDY - 45693242 - DATA-SCIENCE - PORTFOLIO-3¶
Portfolio 3 - Clustering Visualisation¶
K-means clustering is one of the simplest and popular unsupervised learning algorithms. Typically, unsupervised algorithms make inferences from datasets using only input vectors without referring to known, or labelled, outcomes. This notebook illustrates the process of K-means clustering by generating some random clusters of data and then showing the iterations of the algorithm as random cluster means are updated.
We first generate random data around 4 centers.
In [159]:
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
%matplotlib inline
In [160]:
center_1 = np.array([1,2])
center_2 = np.array([6,6])
center_3 = np.array([9,1])
center_4 = np.array([-5,-1])
# Generate random data and center it to the four centers each with a different variance
np.random.seed(5)
data_1 = np.random.randn(200,2) * 1.5 + center_1
data_2 = np.random.randn(200,2) * 1 + center_2
data_3 = np.random.randn(200,2) * 0.5 + center_3
data_4 = np.random.randn(200,2) * 0.8 + center_4
data = np.concatenate((data_1, data_2, data_3, data_4), axis = 0)
plt.scatter(data[:,0], data[:,1], s=7, c='red')
plt.show()
1. Generate random cluster centres¶
You need to generate four random centres.
This part of portfolio should contain at least:
- The number of clusters
kis set to 4; - Generate random centres via
centres = np.random.randn(k,c)*std + meanwherestdandmeanare the standard deviation and mean of the data.crepresents the number of features in the data. Set the random seed to 6. - Color the generated centers with
green,blue,yellow, andcyan. Set the edgecolors tored.
In [161]:
std = data.std()
mean = data.mean()
np.random.seed(5)
claculating the standard deviation and mean from the data with a random seed of 5 and can be further used to generate four random centers¶
In [162]:
k = 4
ss = ['green', 'blue', 'yellow', 'cyan']
setting the no of clusters to 4¶
In [163]:
centres = np.random.randn(k,2)*std + mean ### centroids
generating the four random centers¶
In [164]:
plt.scatter(data[:,0], data[:,1], s=7, c='black')
plt.scatter(centres[:,0], centres[:,1],s=100, c=ss)
plt.show()
plotting the four random centers with the rndom data that we have generated¶
In [165]:
print(data)
In [166]:
d1 = pd.DataFrame({
'X_value': data[:, 0],
'Y_value': data[:, -1]
})
In [167]:
print(d1.head())
In [168]:
print(centres)
2. Visualise the clustering results in each iteration¶
You need to implement the process of k-means clustering. Implement each iteration as a seperate cell, assigning each data point to the closest centre, then updating the cluster centres based on the data, then plot the new clusters.
Replace this text with your explaination of the algorithm. The resulting notebook should provide a good explanation and demonstration of the K-means algorithm.
In [169]:
centroids = {
1: [centres[0][0], centres[0][1]], 2: [centres[1][0], centres[1][1]], 3: [centres[2][0], centres[2][1]], 4: [centres[3][0], centres[3][1]]
}
Converting the random centers into a dictionary with set of key value pair¶
In [170]:
# Assignment Stage
def assignment(df, centroids):
for i in centroids.keys():
# sqrt((x1 - x2)^2 - (y1 - y2)^2)
df['distance_from_{}'.format(i)] = (
np.sqrt(
(df['X_value'] - centroids[i][0]) ** 2
+ (df['Y_value'] - centroids[i][1]) ** 2
)
)
colmap = {1: 'green', 2: 'blue', 3: 'yellow', 4: 'cyan'}
centroid_distance_cols = ['distance_from_{}'.format(i) for i in centroids.keys()]
df['closest'] = df.loc[:, centroid_distance_cols].idxmin(axis=1)
df['closest'] = df['closest'].map(lambda x: float(x.lstrip('distance_from_')))
df['color'] = df['closest'].map(lambda x: colmap[x])
return df
Using the Sqrt((x-x1)2 + (y-y1)2) which is euclidian distance we can calculate the points that are closer to the generated centroids¶
In [171]:
df = assignment(d1, centroids)
print(df.head())
sss =[]
for i in range(0,800):
sss.append(df['color'][i])
Plotting the 800 random points with the color that is appropriate with respect to the closeness with the centroids¶
In [172]:
fig = plt.figure(figsize=(5, 5))
plt.scatter(df['X_value'], df['Y_value'], color=df['color'], alpha=0.5, edgecolor='k')
colmap = {1: 'green', 2: 'blue', 3: 'yellow', 4: 'cyan'}
for i in centroids.keys():
plt.scatter(*centroids[i], color=colmap[i])
In [173]:
#update stage
import copy
old_centroids = copy.deepcopy(centroids)
def update(k):
for i in centroids.keys():
centroids[i][0] = np.mean(df[df['closest'] == i]['X_value'])
centroids[i][1] = np.mean(df[df['closest'] == i]['Y_value'])
return k
centroids = update(centroids)
we re calculate the new centroids from the older centroid and updating¶
In [174]:
suhash = 2
In [175]:
fig = plt.figure(figsize=(20, 10))
ax = plt.axes()
plt.scatter(df['X_value'], df['Y_value'], color=df['color'], alpha=0.5, edgecolor='k')
for i in centroids.keys():
plt.scatter(*centroids[i], color=colmap[i])
for i in old_centroids.keys():
old_x = old_centroids[i][0]
old_y = old_centroids[i][1]
dx = (centroids[i][0] - old_centroids[i][0]) * 0.75
dy = (centroids[i][1] - old_centroids[i][1]) * 0.75
ax.arrow(old_x, old_y, dx, dy, head_width=0, head_length=0.75, fc=colmap[i], ec=colmap[i])
plt.show()
This is the first itteration where based on the colored groups we now calculate the new centroids for each group¶
In [176]:
#Repeat Assignment Stage
df = assignment(df, centroids)
# Plot results
fig = plt.figure(figsize=(20, 10))
plt.scatter(df['X_value'], df['Y_value'], color=df['color'], alpha=0.5, edgecolor='k')
for i in centroids.keys():
plt.scatter(*centroids[i], color=colmap[i])
plt.show()
the figure obve is the second iteration of the kmeans clustering, we can see that the some of the surrounding points were turned to new color¶
In [177]:
# Continue until all assigned categories don't change any more
while True:
closest_centroids = df['closest'].copy(deep=True)
centroids = update(centroids)
df = assignment(df, centroids)
suhash += 1
if closest_centroids.equals(df['closest']):
break
fig = plt.figure(figsize=(20, 10))
plt.scatter(df['X_value'], df['Y_value'], color=df['color'], alpha=0.5, edgecolor='k')
for i in centroids.keys():
plt.scatter(*centroids[i], color=colmap[i], s=100)
plt.show()
print("\n")
print("It took " + str(suhash) + " Itterations in order to get the centroids position to be constant")
It took a total of "six" Iterrations in order for the centroids to be constant¶
In [178]:
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
from sklearn.datasets.samples_generator import make_blobs
from sklearn.cluster import KMeans
In [179]:
ssss = ["cyan","blue","green", "yellow"]
kmeans = KMeans(n_clusters=4, init='k-means++', max_iter=300, n_init=10, random_state=0)
pred_y = kmeans.fit_predict(data)
plt.scatter(data[:,0], data[:,1])
plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1], s=300, c=ssss)
plt.show()
In [1]:
import json
import os
In [2]:
notebooks_to_merge = [file for file in os.listdir(os.getcwd()) if file.endswith('.ipynb')]
notebooks_to_merge.sort()
print(notebooks_to_merge)
In [8]:
def combine_ipynb_files(list_of_notebooks, combined_file_name):
with open (notebooks_to_merge[0], mode = 'r', encoding = 'utf-8') as f:
a = json.load (f)
for notebook in notebooks_to_merge[1:]:
with open (notebook, mode = 'r', encoding = 'utf-8') as f:
b = json.load(f)
a['cells'].extend (b['cells'])
with open(combined_file_name, mode='w', encoding='utf-8') as f:
json.dump(a, f)
print('Generated file: \"{}\".'.format(combined_file_name))
return (os.path.realpath(combined_file_name))
combine_ipynb_files(notebooks_to_merge, "merged.ipynb")
In [ ]: